Working with Bounding Boxes and Polygons
For more details on all the functionality related to bounding boxes, polygons, other regions, in ApertureDB, please refer to our Image ROI documentation and Video ROI documentation
# Dependencies
from aperturedb.Images import Images
from aperturedb.Utils import Utils, create_connector
from aperturedb import NotebookHelpers as nh
from aperturedb.Constraints import Constraints
import os
import helper
# Connect to the ApertureDB instance.
db = create_connector()
utils = Utils(db)
# Parameters used across queries in this notebook.
demo_name = "coco"
data_source = "coco"
width = 600
height = 600
display_limit = 1
label = "horse"
polygon_label = "dog"
roi_label1 = "tie"
roi_label2 = "person"
image_id = 458992
polygon_id = 226510
Display BoundingBoxes using the Python object mapper API
print("\nShowing bounding boxes overlaid on a subset of images from the dataset: " + data_source)
imgs = Images(db, batch_size=display_limit)
const = Constraints()
const.equal("adb_data_source", data_source)
imgs.search(constraints=const)
print("where " + str(imgs.total_results()) + " images matched the query\n")
# It is as simple
# Note: The images are not ordered, and may not have associates bounding boxes.
# those will be diplayed, but not highlighted.
imgs.display(limit=display_limit, show_bboxes=True)
Showing bounding boxes overlaid on a subset of images from the dataset: coco
where 123287 images matched the query
Display Polygons
# Simply switch over to a polygon display
imgs.display(limit=display_limit, show_polygons=True)
Find Images using Labels or Annotations
We use JSON examples here since it allows us to build more complex queries
Retrieve images with a given label being 40% of the image
Graph (metadata) traversal makes it easy to find images and create datasets
print("\nReturning resized version of images with resolution better than: " + str(width) + "x" + str(height) + " with label: " + label + "\n")
q = [ {
"FindBoundingBox": {
# this is key to traversing the graph-based metadata. We are declaring a reference.
"_ref": 1,
"with_label": label,
# ensuring that the box with label is in a given area of the image
"in_rectangle": {
"x": 0,
"y": 100,
"width": 300,
"height": 500
},
# we would like to see the coordinates of this box
"coordinates": True,
# but not retrieve the pixel area contained within
"blobs": False,
"labels": True,
}
}, {
"FindImage": {
# we can use the connection between bounding box and image to go back and forth
"is_connected_to": {
"ref": 1,
},
"uniqueids": True,
# fetch only the images that have a decent resolution
"constraints": {
"width": [">", width],
"height": [">", height]
},
# also apply some operations on the fly to matching images
"operations": [
{
"type": "resize",
"width": 224,
"height": 224
}
],
"results": {
"limit": display_limit
}
}
} ]
responses, blobs = db.query(q)
#Let's make a wrapper with the response from db.query
images_wrapper = Images(db, batch_size=display_limit, response=responses[1]["FindImage"]["entities"])
# and display the results with bounding boxes drawn on top
images_wrapper.display(show_bboxes=True)
Returning resized version of images with resolution better than: 600x600 with label: horse
# Show the above again, with polygons instead of bounding boxes
images_wrapper.display(show_polygons=True)
Doing a LOT more with Intersection over Union
Filter and sort regions by their properties
Polygon regions can be queried based on their properties just like any other entity type in ApertureDB. documentation
This example is looking for high-resolution images of the chosen label.
It queries for polygon regions with the expected label, and displays them in descending order by area (measured in pixels).
images, large_dog_polygons = helper.find_polygons_by_property(db, with_label=polygon_label, property="_area", sort="descending", limit=display_limit)
print("displaying the {} largest {} polygons".format(display_limit, polygon_label))
images.display(polygon_constraints=large_dog_polygons, polygon_tag_key="_area", polygon_tag_format="{:.0f} pixels")
displaying the 1 largest dog polygons
Find overlapping polygons
ApertureDB can be used to identify intersecting polygon regions within an image.
This example is looking for overlap between specified roi_label1 and roi_label2.
Basically the query is for any polygon labeled roi_label1 that intersects with a polygon labeled roi_label2.
print("\nReturning images with annotated polygon: " + roi_label1 + " overlapping with polygon: " + roi_label2 + "\n")
images, overlapping_polygons = helper.find_overlapping_polygons(db, roi_label1, roi_label2)
images.display(polygon_constraints=overlapping_polygons, limit=display_limit)
Returning images with annotated polygon: tie overlapping with polygon: person
Find polygons in a fixed location
ApertureDB supports querying for polygon regions based on their location within an image.
For instance, suppose you want to search a security camera feed for people using a specific door, or to flag objects that may extend out of frame.
This example is looking for the given polygon_label extending out of frame past the upper left corner.
It uses a custom "r" shape to describe the upper-left corner of an image (displayed in red), and then queries for polygons labeled polygon_label that intersect that shape.
upper_left_corner_region = [ # This describes the following shape:
[0.0, 0.0], # ____________
[100.0, 0.0], # | _______/
[95.0, 5.0], # | /
[40.0, 5.0], # | |
[5.0, 40.0], # | |
[5.0, 95.0], # | |
[0.0, 100.0], # |/
]
print("\nReturning images where annotated polygon: " + polygon_label + " spans the upper left corner of the image\n")
images, trains_in_upper_left = helper.find_polygons_within_region(db, with_label=polygon_label, within_region=upper_left_corner_region)
images.add_polygon_overlay([upper_left_corner_region],(0,0,255))
images.display(polygon_constraints=trains_in_upper_left, limit=display_limit)
Returning images where annotated polygon: dog spans the upper left corner of the image
Evaluating an Object Detector by its RoI quality
With ApertureDB, it is easy to compare two sets of regions that describe the same scene.
In this example, we evaluate the similarity between this dataset's polygons and bounding boxes within the same image.
It does this by computing the pairwise intersection-over-union (IoU) values between the individual bounding box and polygon regions. documentation
print("\nIoU matrix for an image with id : " + str(image_id) + "\n")
image = helper.print_iou_matrix_for_image(db, image_id)
image.display(show_polygons=True)
image.display(show_bboxes=True)
IoU matrix for an image with id : 458992
IoU matrix for polygons vs. bounding boxes in image 458992:
PLGN \ BBOX|person| fork |knife |pizza
person|*0.6414|0.0806|0.0208|0.0167
fork |0.0082|*0.1014|0.0000|0.0047
knife |0.0080|0.0000|*0.5920|0.0044
pizza |0.0070|0.0020|0.0000|*0.7717
Polygon Levels of Detail
Polygon regions in ApertureDB can be retrieved at different of detail levels suitable for a variety purposes.
FindPolygon can be used to retrieve polygon geometry at the following levels of detail, from least to most complex:
bounds: Returns the smallest axis-aligned bounding box that contains the region.hulls: Returns the smallest convex polygon that contains the region.vertices: Returns the full-resolution polygon geometry exactly as it was added.decompositions: Returns a breakdown of the region into disjoint convex pieces.
This examples shows all available levels of detail for a single polygon.
print("\nPolygon geometry for a specific polygon with id : " + str(polygon_id) + "\n")
helper.display_polygon_levels_of_detail(db, polygon_id)
Polygon geometry for a specific polygon with id : 226510
